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Introduction 

Gaining entrance into the college of one’s choice is among the most 
exciting and important events in a young person’s life. Consequently, when 
one considers the major impact college entrance exams have in the 
admissions process (Becker, 1990; Educational Research Service, 1981; 
Powers, Alderman, & Noeth, 1983; Wilder, 1989) performing well on these 
exams becomes a high stakes process (Powers, Alderman & Noeth, 1983; 
Wilder, 1989). Because of the importance of these tests, many students 
and parents are prepared to pay large sums of money for test preparation 
classes (Becker, 1990, Smyth, 1989, Wilder, 1989). Preparation courses 
for the SAT are booming, with as many as 100,000 to 150,000 students out 
of 1.8 million SAT takers participating (Putka, 1992). In addition, many 
schools are including test preparation in their curriculum. It is estimated that 
as many as one-third of the schools in the Northeast offer some type of 
preparation course. This interest in specialized test preparation, 
sometimes referred to as “coaching”, raises questions both about its 
effectiveness in raising scores and its ethics, since the cost may exclude 
many students (Putka, 1992) and methods (Becker, 1990; Educational 
Research Service, 1981 ; Popham, 1991). 

Over the past 30 years interest in the effectiveness of coaching has 
produced dozens of studies, almost all directed toward what is now called 
the Scholastic Assessment Test (SAT) (Becker, 1990; Hulsart, 1983, 
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Powers, 1982). There appear to be two major reasons why the research 
has focused on the SAT : 1) The SAT is used by schools where the 
competition is the greatest; and 2) the SAT has over time evolved into a 
barometer used to measure the academic effectiveness of high schools 
(Hulsart, 1983; Smyth, 1989). 

Although the American College Test (ACT) is also taken by a large 
number of students, it has been less widely publicized (Hulsart, 1983) and 
researched. Doctoral research completed by this author in 1993 
investigated the effects of coaching on the ACT scores of students at a 
large Midwestern university. A complete search of the ERIC, PSYCHLIT, 
SSCI and Dissertations Abstracts International produced only 2 other 
published research documents examining the effectiveness of coaching for 
the ACT. 



Purpose of this Research 

Many colleges and universities have initiated programs to increase 
the enrollment of African-American students. Most of these programs 
provide some preparation for the standardized admissions test these 
students need to take. For high school students in the city of St. Louis, the 
majority of whom ar African-American, the test of choice is the ACT. 

Since little published research exists about the effectiveness of 
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coaching on increasing ACT scores , the purpose of this research was to 
investigate whether ACT preparation provided by professional coaches 
could increase the scores of African-American students. If coaching can 
significantly raise the scores of African-American students, their access to 
institutions of higher education, especially the more selective universities, 
can be improved. And although the net result is the same, the implicit 
benefit of improving college entrance tests scores is that college access is 
increased by improved test scores rather than lowered standards, the 
practice now utilized by some institutions who are eager to increase their 
enrollment of African-American students. 

Perspective or Theoretical Framework 

A student’s score on a standardized test can be considered to have 3 
components; a true score component, a test specific component, and 
random error (Becker, 1990). Both the true score and test specific score of 
an individual’s score are targets of coaching interventions. 

Most coaching interventions can be classified into four categories 
(McCormack, 1987). The first type of coaching seeks to improve general 
testwiseness. The second attempts to increase test familiarization, 
targeting the test-specific score. This style of coaching has a relatively 
short duration and is designed to acquaint students with the formats of the 
questions on a specific standardized test. The third style can best be 
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described as “concentrated drill and practice” on sample items likely to be 
similar to those found on actual tests. This style of coaching also targets the 
test-specific component. The fourth type of coaching involves instruction in 
the content domain. This style would include coaching which provides a 
systematic review of the content of high school courses related to a specific 
standardized test. This type of intervention takes by far the most time and is 
concerned with increasing the domain knowledge of the student. This 
coaching strategy is designed to increase the overall knowledge and hence 
seeks to increase an individual’s “true score.” 

Some research demonstrates that each of these four coaching 
interventions may be effective in increasing scores on standardized tests. 
The following sections summarize some of the research for each 
intervention practice. Since there is so little research about ACT 
preparation, most of the researched summarized will deal with the SAT. 

The Effects of Coaching to Improve General Testwiseness 

Some students prepare for the SAT by receiving instruction in general 
problem solving strategies. The meta-analysis by Kulik, Bangert-Drowns, 
and Kulik (1984) revealed mixed results. Their analysis showed that 2 
studies which included this type of coaching had an average effect size 
(treatment vs control) of .13 (standard deviations). This is compared to 1 1 
studies which were only drill on past tests (effect size = .18 standard 
deviations) and one study which just provided a short orientation (effect 
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size = .05). 

Kulik, Bangert-Drowns, and Kulik (1984) also compared studies which 
included a testwise component (n=5 studies) to those which did not (n=9 
studies). Their analysis revealed an average effect size with this 
component was .20, while the effect size for those without was .08. This 
would indicate that instruction in testwiseness is effective in increasing SAT 
scores. 

Relative to other types of coaching interventions, there has not been 
much research regarding the effects of increased testwiseness on scores 
on the SAT. In general, Walstrom & Boersman (in Anastasi, 1981) found 
that short orientation sessions can be effective in equalizing the differences 
between student’s testwiseness. 

Anastasi (1981) believes that the relatively small gain attributable to 
testwiseness is due to the amount of standardized test sophistication most 
students develop before they take the SAT and that all that is required for a 
modest improvement is a short orientation to the specifics of a particular 
standardized test. 

Summary of the Research to Increase Test Familiarity 

A study by Powers and Alderman (1979, in Powers, 1982) revealed 
that students who read “About the SAT,” a short booklet designed to orient 
students to the SAT, felt that it helped them improve their scores. A 
comparison of the actual SAT scores, however, revealed little gain that 
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could be attributed to this preparation. Most research concerning the SAT 
groups this intervention with specific item practice. Very little research has 
been conducted relative to the effects of providing students with specific 
strategies for specific scores on specific tests. 

Summary of the Research on the Effects of Specific Item Practice 

Much of the coaching that is received by students trying to improve on 
the SAT involves practicing on items similar to those which they will 
encounter on the actual test. Since the Educational Testing Service (ETS) 
publishes past forms of he SAT, most of this practice is with former SAT 
questions. 

In her meta-analysis, Becker (1990) examined the effects of practice 
with specific test items as well as practice using complete forms of past 
tests. Her results revealed that practice with specific items was the single 
most Important variable associated with improving scores on the SAT. 
Interestingly, she found no relationship between taking complete past tests 
and improvement in SAT scores-suggesting that coaches who can identify 
specific types of items and provide practice with these items are effective in 
improving SAT scores. 

As part of their meta-analysis, Kulik, Bangert-Drowns, and Kulik 
(1984) compared the effect size of studies which included practice (n= 13 
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studies) with those which did not (n=1). Their results revealed an average 
effect size of .16 (standard deviations) for the studies with practice and an 
effect size of only .03 for the study without. One problem with this result, 
however, is that there is no explanation about the type of practice that was 
provided by these studies. Another problem, of course, is that there was 
only one study that lacked a practice component. 

In their study of urban gifted students Reynolds, Oberman, and 

Perlman (1988) also found practice with specific items to be effective in 

improving SAT scores. In their conclusion they write: 

Statistically and practically significant gain scores can be obtained on 
the SAT and PSAT, especially with the use of old tests for practice, 
drill, and feedback. The use of past PSAT forms serves the relevant 
function of familiarizing students with actual PSAT items rather than 
approximate items, thus giving students a greater sense of the type of 
items they can expect on the test. (p. 162). 

Research on the effects of Long-Term Instruction in the Content Domain 

Some studies have investigated the effects of long-term Instruction in 
the content covered by a standardized test. Gains achieved through such 
intervention are sometimes referred to as “alpha” gains (Bond, 1989). The 
results of the meta-analysis by Becker (1990) revealed no relationship 
between this type of coaching and improved SAT scores. However, since 
the ACT is more of an achievement test than the SAT, this may be one area 
in which there is a large difference in the effectiveness attributable to a 
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particular intervention on the ACT. 



Powers (1988) found a systematic review of the content domain was 
effective if the students were coached, but ineffective for those who were 
not. 

Summary of the Research on the Effects of Coaching on ACT Scores 

This section summarizes three research documents that investigate 
the effects of coaching on ACT scores. These are the only studies found 
after a complete search of the ERIC, PSYCHLIT, SSCI, and Dissertations 
Abstract International data-bases. Since the ACT underwent a major 
revision in 1989, only studies since 1989 were included in this search. 

The first study by Lauderdale (1989) investigated the effects of using 
microcomputers and Krell software (a commercial software package 
designed to give students practice with simulated ACT items) as tools for 
preparing for the ACT. This study found no significant gains as a result of 
using Krell software and microcomputers (Lauderdale, 1989). 

The second study by Seaton (1992) investigated the effects of 
coaching on ACT scores of 30 juniors from a large urban high school for 
girls. This study revealed that students who received a 1 0 hour preparation 
course increased their composite scores an average of 4 points (from 19 to 
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23, t=7.2, p<.001). For this study, the pretest was a retired ACT test, while 
the post test was an actual. ACT. Consequently, there might have been a 
motivational difference which also contributed to this four point gain. 

The third study is the doctoral research conducted by this author 
(Moss, 1993). This study investigated the effects of coaching on the ACT 
scores of students at a large Midwestern university. In this study, freshmen 
who had taken the ACT more than once were surveyed regarding their 
preparation for their second ACT. This study found no significant difference 
between the gain for students who said that they had received coaching, 

1 .09 points (n=52) and those who did not, .94 (n = 441). It is important to 
note, however, that this study did not investigate whether coaching can be 
effective, but whether or not the coaching these students received was 
effective. Taken together, these three studies indicate that how a student is 
coached may be more important than JLthey they are coached. 
Unfortunately, the study conducted by Seaton (1992) did not give any 
indication as to the types of coaching interventions employed. 

Sample 

The students in this study were all law interns, a special summer 
program conducted for juniors and seniors by St. Louis Public Schools. 

This program is part of a larger project, “The High School-Post Secondary 
Transition Initiative,” funded through the Division of State and Federal 
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Programs. One goal of this program is to help more students from St. Louis 
Public Schools gain admission to college. To help achieve this goal, 
preparation for the ACT is provided to any student on a voluntary basis. 
Although there were over 100 interns in the program, only 37 chose to 
participate in the coaching class. Out of these 37, 21 took both the pretest 
and the post test. Two of these students were not African-American, thus 
leaving 19 African-American students in the sample. 

Design 

On Saturday, June 23, 1994, all students who participated in the coaching 
class were given ACT 9340D as a pretest. This pretest was published by 
ACT (1993) and is comparable to actual tests. The test was administered 
using the same procedures that are employed during actual ACT testing. 

On the next four Saturdays, each participating student received three and 
one-half hours of instruction. This instruction was provided by professional 
coaches from Focus on Learning, a private tutoring company in Columbia, 
Missouri, and utilized the four coaching interventions previously identified. 
Each of the coaches was a specialist in the content domain being instructed 
and was experienced with coaching the specific strategies relevant to the 
ACT. 



Both the English and the mathematics sections utilized a three step 
process. The first step was a systematic review of the content relevant to 
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these two tests. The second step focused on general and specific 
strategies, with the majority of the time spent discussing specific strategies 
for these two sections. A key specific strategy that was stressed was how to 
pace oneself in order to achieve a predetermined goal score. The third step 
provided students practice with real ACT items and offered specific 
feedback regarding their performance. 

Since the content on the reading and science reasoning tests are not 
as clearly defined as the content on the English and mathematics tests, 
preparation for these two tests involved only steps 2 and 3 (strategies and 
practice using past ACT questions). 

In order to increase the potential effectiveness of this class, the 
students were divided into either the “high” or “low” group based on their 
pretest scores. Each group received 1 3/4 hours of English/reading 
instruction and 1 3/4 hours of math/science instruction each Saturday. 

On the Saturday following the four weeks of instruction, each student 
was given another retired ACT (form 9139C) as the post test. In addition, 
each student was asked to complete a questionnaire regarding previous 
preparation, how many times they had taken a real ACT prior to the 
program, and how motivated each was to improve. These variables served 
as control variables within the analysis. Demographic information such as 
gender and the level of parent’s education were also collected. These 
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allowed for the investigation of other factors potentially related to the 
effectiveness of preparation. 

Although it would have been better to randomly select the students for 
this preparation and to have had a randomly selected control group, this 
was not possible for this study. The funding for this class required that all 
students who wanted to receive this preparation be included In the class. 
This was an unavoidable limitation in this study. Consequently, there may 
be some self-selection bias affecting the level of motivation to prepare for 
the test. 



Results 



The results of this study are summarized by table 1 



Table 1 



Pretest and posttest means 



Section 


Pretest 


Posttest 


Gain 




mean 


SD 


mean 


SD 




English 


15.05 


3.89 


16.79 


4.20 


1.79^^ 


Math 


16.58 


4.72 


17.37 


4.28 


0.79 ^ 


Reading 


16.26 


4.83 


17.26 


4.63 


1.00 ^ 


Science 


16.05 


3.70 


17.89 


4.05 


1.84^ 


Composite 


15.98 


3.73 


17.33 


1.34 


1.34^^ 


n= \9;*p < .05; 


** p < .( 


)1 
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Since each of the tests are conceptually different, separate 
dependent t-tests were conducted to analyze the gain on each test as well 
as on the composite score. Regression analysis revealed no significant 
effects for any of the control variables or demographic variables. This, 
however, may be due to the relatively small sample size, hence making 
these low power tests. 



Discussion 

There are several factors to which the gain in ACT scores might be 
attributed. There was probably a slight practice effect, although since many 
of the students indicated that they had already taken at least one real ACT, 
this should have been minimal. Another possible factor may have been 
maturation. However, since this was only a 6 week program, this also 
should have been minimal or even negative since the preparation took 
place while the students were not in school. Third, since the students who 
dropped out scored, on average, lower than those who stayed, regression 
toward the mean would have actually pulled the gain down slightly. 
Moreover, the students had very little motivation to improve, most indicating 
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that they had not put any time into studying outside of the class. 



Consequently, the great majority of the increase in ACT scores 
showed by these students can be attributed to the coaching they received. 
And although the average increase was only a modest 1 .34 points, it could 
make a difference in obtaining admission for those who benefited most. 

Educational or Scientific Importance 

This study shows that, with proper interventions, the ACT scores of 
African-American students can be increased. Since other studies have 
found a positive effect for duration (Messick & Jungeblut, 1981) it is highly 
probable that more student-coach contact time (14 hours for 4 subjects is 
relatively short) would increase the effect of preparation. This would seem 
to be especially true if students are preparing for a “real” ACT during the 
school year. (Other unpublished research by this author has found that the 
timing of the coaching makes a difference to the outcome). Consequently, 
providing African-American students with appropriate intervention during 
the school year can increase their scores ad increase their access to 
institutions of higher learning. 

Implication for Policy or Practice 

The results of this study would suggest that all African-American 
students who plan to go to college should have access to appropriate 
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preparation for the ACT. Moreover, since students who are currently 
receiving professional preparation are disproportionately white, providing 
coaching to all African-American students may help bridge the gap between 
the ACT scores of these two races. 

Need for Further Research 

Further research investigating the effects of coaching on the ACT 
scores of African-American students needs to be conducted. This research 
should use a larger sample and, if possible, random selection, and a control 
group. A larger sample would allow for an examination of the specific 
effects of. each type of intervention for specific sub populations (e.g. girls or 
low-ability students). Random selection would facilitate the generalizability 
of the study to all African-American students. 
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